Repository navigation
Merge the 26.11.1.post4 release line into main - #410
Merged
Merged
Conversation
… from an int64 buffer; the stored vertices match the JSON route (refs #388, #315) Stub-library tests (both backends) for the routing, the header, the chunking, the RID text and every shape that keeps the JSON route; a native-only parity test that loads the same rows with the route on and off and compares the stored records, their property order and the RIDs. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…the RIDs in an int64 buffer (refs #388, #315) arcadedb_n_gb_colp takes the address table of arcadedb_n_insert_colp, builds the rows with the objects and the property order the JSON rows give (BulkOps.Table.cell, BulkOps.vertexRows) and calls the same GraphBatch.createVertices(type, Object[][]); (bucket, position) of each new vertex are written into a buffer of the caller. Rows with other keys or another key order, a nested value, a mixed column, a set, a numpy scalar, an integer past 64 bits, a lone surrogate or a pandas Timestamp with nanoseconds keep the JSON route. ARCADEDB_NATIVE_GB_COLP=0 turns it off for A/B runs. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nd get_rid of a record are one call each on the JVM build (refs #388) The tests count the calls on a proxy of the Java row or record and of ReaderCalls: the parent makes four calls for Result.get_rid, three for each element getter and two for Document.get_rid, and none on ReaderCalls. A second set compares every answer with the per-call path for a vertex, an edge, a document, a projection and an aggregate. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…et_rid cross once each, through ReaderCalls (refs #388) ReaderCalls.ridOf, vertexOrNull, edgeOrNull, elementOrNull and documentRid answer in one crossing what took getIdentity, isPresent, get and toString for a row id, getVertex/getEdge/getElement, isPresent and get for an element, and getIdentity plus str() for a record id. A record with no identity yet still reads as the text str(None), as before. The wheel carries its own bridge jar, so the per-call route is gone from these five methods. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ach record keeps its own callbacks (refs #388, #315) The records of one type cross as one columnar frame with the callback ids of each record, not one JSON op per record. The recorded-library tests count crossings and pin the order, the copy of the properties at the call, the shapes that keep the per-record op and a record that is refused in the middle of a batch; the native tests compare the stored values with the per-record op and run the real callbacks. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nar frame with per-record callbacks (refs #388, #315) create_record collects the records of one type and kind (a copy of their properties) and hands _ASYNC_BATCH of them over at once through arcadedb_n_async_bin, op create_records, with the callback ids of each record. Every other call on the executor, wait_completion, close and kill hand the collected records over first, so the engine queue keeps the order of the calls. A shape the frame cannot carry and a batch of one keep the per-record op. A record that is refused before it is queued reaches its own error callback, or is raised by the call that hands the batch over, after the records around it were handed over. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ssing numpy fails and never skips (refs #388) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…atements, so Bandit B608 has nothing to flag (refs #388) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ch frame routing test (refs #388, #315) A patch of the first commit never applied (the test runner call in the same command was blocked), so the file kept: a schema row for a type with an INTEGER property while the test sent numpy int32 scalars (a list of numpy scalars is no column: the JSON route, correctly), a stand-in that read body["rows"] for the count form (KeyError), and a 21-row load in chunks of 10 that made two chunks where the test expected three. None of these was JVM-specific and none was a fault of the route. The declared-property test now sends a declared ARRAY_OF_FLOATS list column, the stand-in reads the count form, the chunk test has three chunks with the odd key in the middle one. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nc executor between phases, and a collected record is handed over without another call (refs #388) CI of 89c44d9: close() shuts the engine's async executor down for good, so the second phase of each parity test failed with 'Async executor has been shut down'; DELETE needs a transaction. The new linger tests pin that a record created with no other call on the executor reaches the library after the linger time, that a failure of that hand-over is raised by the next call, and that close() stops the thread. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…other call on the executor (refs #388) test_async_executor.py::test_per_record_callback_runs_before_the_batch_commits_and_on_ok_after creates one record and waits for its callback, with no other call on the executor, so the record sat in the Python buffer for ever. A thread of the executor (started with the first collected record, stopped by close()) hands a batch over when its first record has waited _ASYNC_LINGER; a failure of that hand-over is raised by the next call that hands records over. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…properties, and only this route refuses Timestamps with nanoseconds (refs #388, #315) Two faults of the first commit that the native parity run showed: 1. Property order. The JSON route took float32 array columns out of the rows (take_vector_columns) and appended them after the other properties (NativeExtras.withVectors), so a vertex of {i, v} was stored [i, v], while the frame route stored the HashMap order [v, i]. The header now carries "last" (the float32 array columns), BulkOps.vertexRows puts them after the HashMap-ordered rest, in column order. Declared list-of-float columns stay in the HashMap order, as in the JSON rows (the parity shapes 'declared vectors from lists' and 'from float32 arrays' cover both). 2. Timestamps. The nanosecond guard sat in the shared packer and so changed insert_many, whose JSON route keeps microseconds (test_native_columnar_temporal.py::test_pandas_timestamps_keep_microseconds_as_the_json_path_does and test_native_mixed_chunk.py::test_a_pandas_timestamp_is_stored_as_the_statement_path_stores_it, red since that commit). pack() and _list_column() take exact_nanos (default off); only create_vertices_columns sets it, because GraphBatch's JSON route writes isoformat() and keeps the nanoseconds. Own tests fixed: 'mixed int and float' with one row is a column of one float and is carried; the two refusals differ only by the type name in the message; the pack test now pins both packer modes and the 'last' header; new end-to-end test of the paper shapes (Person rows in chunks, Product rows with float32 vectors: only arcadedb_n_gb_colp crossings, no create_vertices JSON crossing, no codec hook, vectors stored bit-equal). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…es follow the new route (refs #388, #315) test_native_ingest_defaults.py (3 tests: default commit_every is 1000, explicit commit_every 250 and 0): asserted the rows per create_vertices crossing [1000,1000,500,1000,1000,500], [250]*10 and [2500] by wrapping NativeGraphBatch._op. Plain rows now cross as columns and do not pass through _op, so the spy also records len(rows) at bulk.create_vertices_columns; the expected numbers are unchanged (the chunking is the same). test_native_graph_batch_binary_vectors.py (7 tests) and test_native_vertex_vector_crossings.py (6 tests): asserted that create_vertices of float32 vector rows crossed arcadedb_n_gb_bin (or arcadedb_n_gb for the text arm) with the side buffer, the "vecs" entry and no per-row codec hook. A plain vector load now crosses arcadedb_n_gb_colp (tests/test_native_graph_batch_frame*.py assert that, and the stored bits). These files' subject is the side-buffer route, which a chunk the columns cannot carry (a nested value, ragged vectors) still takes, so their db fixture turns the columnar route off (lib.GB_COLP False); no assertion is loosened. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… values, not the order of the properties, and the refused records are proved by what is stored (refs #388) The parity run on the installed wheel failed 16 of 21 tests of test_native_async_create_batch.py on every platform. The values of the batch and of the per-record op were equal in all 14 shapes that were compared; two facts of the real library that the test had guessed wrong: - The per-record op parses the properties of a record into a hash map (Json.java), so it stores them in hash order ({v, i} for the call {i, v}), and the batch stores them in the order of the call. The order of the properties of a document is no promise, so the dump sorts the keys of each row. - The writer reports a record stored once it is in the transaction; a refusal by a unique index comes at the commit, after that. The records 4 and 7 of the refusal test reach their callback and then their error callback, on both routes (the equal results say so). The test now needs the callback of every stored record and checks the stored (key, index) pairs. A record with numpy scalars is not a shape the column reader carries (Python scalars only, as for insert_many), so it crosses as the per-record op; the test lists it with the other shapes without a frame and the docstring of create_record names it. The shape key_order is renamed unsorted_keys. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…py passes an importer setting that the latest engine refuses; refs #388) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…keep the plain path when the bridge class is unavailable (refs #388) The tagged JVM run (26.11.1, 43590179526) failed two tests of test_reader_crossings.py on every platform leg with AttributeError: 'NoneType' object has no attribute 'elementOrNull'. Both put None under _BRIDGE_CLASSES["ReaderCalls"], which is what _bridge_class answers when the bridge jar is missing, to read a record by the per-property path. The new readers called ReaderCalls without the None check every other reader of the file has, so a missing bridge class raised instead of falling back. Result.get_rid, Result.get_vertex, Result.get_edge, Result.get_element and Document.get_rid now take the old calls (getIdentity or getX, isPresent, get, toString) when _bridge_class gives None, and the one crossing otherwise. A new test runs every kind of row (vertex, edge, document, projection, aggregate) through all five readers with and without the bridge class and expects the same answers; the two failing tests are unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
) The timed A/B of this branch read figure j_rid_docobj_100k 1.6 times slower per call (1.567 us before, 2.558 us after, A/A 1.015, parity ok, the control and the other rid figures unchanged or faster). The plain path is two JPype operations already (getIdentity, then str of the RID); the bridge call is two as well (the static call and the conversion of the String it returns) and the lookup of the bridge class on every call adds to them (about 0.3 us by a plain Python timing of the helper). The Result getters keep their one call: they went from four or three operations to one. ReaderCalls.documentRid and its test are removed or changed to say so. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…route of post3 (refs #388, #401) The laptop A/B of this pull request measured a single row 1.63 times slower than post3 (n_gb_v_1) and batches of 100 rows and more faster: the columnar entry has a fixed cost (a header, the column specs, the buffers) that a handful of rows does not repay. A chunk below lib.GB_COLP_MIN_ROWS (64, ARCADEDB_NATIVE_GB_COLP_MIN_ROWS, 0 turns the floor off) now takes the post3 JSON route; at or above it the columns, as before. Sizes 1 and 100 are measured and the crossover lies between them; below 64 the path is the post3 path, 100 and up is measured faster, 64 to 99 is not measured. The check is in bulk.create_vertices_columns, so a small chunk reads no schema and builds no columns. Tests: the stand-in library of the pack tests turns the floor off so the existing tests keep pinning the columnar route; new tests pin the floor at 1, 63, 64, 65 and 100 rows, that a small chunk builds nothing, that chunks are judged one by one, the switch and the environment variable. The native parity test loads the same rows with the default floor, with the JSON route and with the columns forced, at 1, 63, 64, 65 and 100 rows, and compares every stored record and the RIDs. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e op each, as in post3 (refs #388, #401) The laptop A/B of this pull request measured a single record 1.57 times slower than post3 (n_async_nar_1) and batches of 100 records and more faster. Collecting has a fixed cost per executor (a linger thread, the locks, the copy of the properties, then the columnar frame) that a small load does not repay. An untimed micro of one executor call (a new executor, N records, wait_completion) on the wheel of this branch against post3, native, measured: slower for 1, 4, 8 and 16 records (1.6x, 2.7x, 1.7x, 1.1x), faster for 32, 64 and 100 records (0.74, 0.54, 0.42); the crossover is near 20 records. The first _ASYNC_DIRECT (32) records of an executor now take the post3 path exactly: no copy, no buffer, no linger thread, no frame. From the 33rd record on the records are collected and handed over as frames, as before. An executor of up to 32 records cannot be slower than post3; the pre-registered figures n_async_nar_1 and n_async_nar_100 verify the change. A fallback at flush time was considered and not used: it would not remove the linger thread and the copy, which are most of the cost of a small load. Tests: the existing tests pin the collecting route with the direct phase off; new tests pin that 1, 2, 31 and 32 records send one op each and collect nothing, that 33, 34, 64 and 100 keep every record and their order, that 0 turns it off, and the default; the native parity test stores the same rows with the default and with the collecting route at 1, 31, 32, 33, 64 and 100 records and compares every stored record. The new tests fail when the direct branch is removed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…cords goes one op each, as in post3 (refs #388, #401) The untimed micro of the first fix head (the first 32 records of an executor one op each) read executors of 33 to 64 records 1.2x to 1.7x slower than post3: a columnar frame costs about 270 us per crossing, a record in a frame about 3 us, a per-record op about 12 us. A frame repays itself from about 30 records; the decided floor is 64 collected records at the hand-over, below it the post3 op per record. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nment test, as the repo does (refs #388, #401) The Bandit job of CI (bandit==1.9.4 -c pyproject.toml -r src tests, low severity and confidence) flagged the import and the call of the test that starts a fresh interpreter to read lib.GB_COLP_MIN_ROWS. The call is an argv list with no shell: this interpreter and a fixed one-line program. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e op each, as in post3 (refs #388, #393, #401) The first fix head collected from the 33rd record and read 1.2x to 1.7x slower than post3 for executors of 33 to 64 records; the frame floor of 64 left 1.2x to 1.3x. The direct phase is now 96 records, so an executor of up to 96 records is exactly post3 and the frame (from 64 collected records) serves executors beyond about 160 records. An executor of 97 to 159 records still pays the per-executor collection cost without a frame (1.1x to 1.2x slower in the untimed micro); a linger thread shared by the whole process would remove it and is a follow-up. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ne op each, as in post3 (refs #388, #393, #401, #406) A direct phase of 96 put the band of the collection cost on the 100-record case: the untimed micro read 1.19 at 100 records, 1.12 to 1.19 for 97 to 159. The cost of the band is a near-constant amount per executor, about 0.2 ms, so its relative size falls as the direct phase grows (about 20 percent at 96, about 15 percent at 128). The band is not removed, it moves to 129 to 191 records (the frame starts at 128 + 64 = 192); a linger thread shared by the whole process is the real fix (#406). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
# Conflicts: # native/src/main/java/org/humemai/nativelib/BulkOps.java # tests/conftest.py
The tag 26.11.1.post4 is built from e83947e; this merge keeps it an ancestor of main. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request must be merged with a MERGE COMMIT, not squashed and not rebased. It brings the release line of
26.11.1.post4intomain, and the release tag26.11.1.post4has to stay an ancestor ofmain; a squash would break that ancestry. The repository allows merge commits.What does this PR do?
I merge
release/26.11.1.post4(e83947e64661) intomainwithgit merge --no-ff. The release line branched from the tag26.11.1.post3. It has 28 commits that are not in26.11.1.post3and 28 commits that are not onmain. Pull requests in the line: #390, #393, #395The merge conflicted in
src/arcadedb/_native/extras.py. The resolution is in the merge commit and touchessrc/arcadedb/_native/extras.py,src/arcadedb/_native/lib.py; it keeps the signature aliases ofmain(Nfor the C int) for the whole table oflib.py.The engine is unchanged: the identity guard compares the CI-built wheels with the released
26.11.1, and the result is in #401.Refs #401, #388.
🤖 Generated with Claude Code